Skip to content

Add Arrcus ArcOS as a containerlab device (best-effort) - #3681

Draft
roc-ops wants to merge 93 commits into
ipspace:devfrom
roc-ops:arcos-device
Draft

Add Arrcus ArcOS as a containerlab device (best-effort)#3681
roc-ops wants to merge 93 commits into
ipspace:devfrom
roc-ops:arcos-device

Conversation

@roc-ops

@roc-ops roc-ops commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Add Arrcus ArcOS as a containerlab device (best-effort)

This PR adds Arrcus ArcOS as a netlab device. ArcOS runs as a native
containerlab node (kind: arrcus_arcos) — there is no vrnetlab packaging and no
Vagrant box. Support level is best-effort.

Connection and validation model — no core change

ArcOS ships the official arrcus.arcos Ansible collection (the project-recommended
network_cli model), but every published version (2.0.13–2.0.18) hangs against the
tested container image: its cliconf/terminal plugins issue a config terminal /
arcos_cli mode-switch that this build's confd_cli rejects. Configuration is
therefore deployed over Ansible's built-in docker connection plugin (docker exec running confd_cli load merge / commit) — the same pattern
sonic_clab already uses. The device keeps a commented-out network_cli block to
switch to once a working collection version ships.

netlab validate reads device state over the same docker-exec path
(show <path> | display json | confd_cli, parsed as OpenConfig JSON by
netsim/validate/**/arcos.py). This is netlab's standard device-side show-command
validation — no ansible validation action and no change to any core file.
netsim/cli/connect.py, netsim/cli/validate.py, netsim/augment/validate.py,
netsim/attributes.yml, netsim/templates/provider/clab/clab.j2, and tests.py are
untouched (byte-identical to dev). This is a pure device addition.

The tested image boots with SSH/NETCONF/gNMI disabled and refuses to enable
interfaces until the factory admin-user password is changed; both are handled with
the native netlab_start_exec group_var (containerlab's post-start exec:) — no
custom deploy code.

Module coverage

initial, ospf (v2/v3), bgp, isis, vrf (+ospf/isis/bgp), vlan, lag,
gateway (VRRP), dhcp/relay, bfd, routing (static/prefix-set/policy),
mpls (LDP), sr (SR-MPLS via IS-IS), srv6, vxlan, and evpn (L2VNI).

Honest limitations are documented in docs/caveats.md (e.g. EVPN symmetric IRB/L3VNI,
VRF route-target leaking, and OSPFv2 area authentication are genuinely unsupported and
not declared; MPLS/SR/SRv6 bring up the control plane only on the native container;
DHCP relay renders but its datapath is not exercised on a clab-only host, so ArcOS is
omitted from the DHCP support table).

Verification

Verified on real ArcOS containers (image arcos:8.2.1A.P2), containerlab provider,
against this branch:

Test Result
02-ospf — OSPFv2 adjacency 2/2 PASS (FULL both directions)
03-bgp — eBGP session + prefixes 4/4 PASS (ESTABLISHED + peer loopbacks in RIB)
04-isis — IS-IS adjacency + reachability 4/4 PASS
05-routing — static route 2/2 PASS; kernel FIB (proto static) installed and loopback-to-loopback ping 0% loss
combined ospf+bfd+vrf deploys clean; per-VRF OSPF reaches FULL; BFD session UP both directions

netlab initial commits every module cleanly (idempotent on re-run) and netlab collect retrieves the running configuration. The rendered CLI and the show/get paths
used by the validators were cross-checked against the vendor ArcOS S8.5.1A CLI
reference.

Files

  • Device: netsim/devices/arcos.yml
  • Deploy/collect tasks: netsim/ansible/tasks/{deploy-config,fetch-config}/arcos.yml
  • Templates: netsim/ansible/templates/*/arcos.j2 (16 modules)
  • Validators: netsim/validate/arcos.py + netsim/validate/{ospf,bgp,isis,routing}/arcos.py
    (adds the netsim/validate/routing/ package)
  • Integration tests: tests/integration/platform/arcos/
  • Docs: docs/platforms.md, docs/caveats.md, docs/install/arcos.md,
    docs/release/26.07.md

This PR also removes netsim/templates/provider/libvirt/arcos-domain.j2, an orphaned
libvirt domain template from a much older, never-completed ArcOS device (no
netsim/devices/arcos.yml ever existed for it); the new device is containerlab-only.

ArcOS is a commercial NOS, so users supply their own container image (as with
sros/dellos10/arubacx).

@ipspace ipspace left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before going into the details: I think you should change the configuration deployment method.

It looks like you're generating a bash script that takes a config and runs confd_cli to deploy it. If that's the case, you don't need Ansible -- netlab can generate bash scripts, map them to the container, and then execute them inside the container. You can also map templates into container files to create scripts (like what you have in the shell command) that are then executed.

FRR uses the "run a bash script" approach together with "let's add a shebang when needed" trick. cRPD maps a template to a container script to generate the deployment script (/config/netlab/netlab-config.sh).

More @ https://netlab.tools/dev/config/deploy/#linux-configuration-scripts

@ipspace

ipspace commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Next major item: we don't have per-platform integration tests (what you added in tests/integration/platform). Please remove those.

The device implementations are tested with regular integration tests (see https://netlab.tools/dev/integration-tests/). If you want to test the validation plugins, you'll find some tests in tests/platform-integration/validate.

Comment thread netsim/ansible/templates/bgp/arcos.j2 Outdated
global afi-safi IPV6_UNICAST
{% endif %}
{% for a in bgp.advertise|default([]) %}
{% if a.ipv4 is defined %}

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more generic comment -- I strongly recommend you use the same indentation trick we use in other Jinja2 templates to make the templates more readable. Unfortunately, that's the best I could come up with, and it looks like some vendors picked it up as well.

For example:

{% for x %}
{%   if y is defined %}
{%   endif %}
{% endfor %}

roc-ops added 2 commits July 22, 2026 19:33
Modernize the legacy ArcOS stub into a full containerlab (arrcus_arcos)
device: initial, ospf (v2/v3), bgp, isis, vrf, vlan, lag, gateway (VRRP),
dhcp/relay, bfd, routing (static/prefix/policy), mpls (LDP), sr (SR-MPLS via
IS-IS), srv6, vxlan and evpn (L2VNI) configuration templates, plus OpenConfig
JSON validation plugins (ospf/bgp/isis/routing).

Configuration is deployed with netlab native "sh" config mode: each module is
rendered into /config/netlab/NN-<module>.sh with a
#!/config/netlab/netlab-config.sh shebang and executed via docker exec, which
hands the rendered config to the mapped wrapper
templates/provider/clab/arcos/netlab-config.j2 to load through confd_cli
(load merge / commit). This is the same mapped-script deploy pattern Juniper
cRPD uses; no Ansible is involved in the deploy path. ansible_connection:
docker is retained only for the validation/collect path, since the tested
build has SSH/NETCONF/gNMI disabled.

Drop the legacy Ansible deploy-config task and the unused libvirt domain
template (the device is clab-only).
Install guide, caveats, platform table entry and release note for the ArcOS
containerlab device, documenting the native sh-mode confd_cli deploy path and
the docker-exec validation model.
@roc-ops

roc-ops commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review, Ivan — all three addressed.

1. Deploy method. Switched the ArcOS device from the Ansible deploy task to netlab's native container-script (sh) config mode, following cRPD: a wrapper netsim/templates/provider/clab/arcos/netlab-config.j2 is mapped to /config/netlab/netlab-config.sh, and each module is rendered with a #!/config/netlab/netlab-config.sh shebang and executed via docker exec, which hands the rendered config to the wrapper to load through confd_cli (load merge/commit). netsim/ansible/tasks/deploy-config/arcos.yml is removed; no Ansible is involved in deployment. I kept the docker connection only for the validation/collect path (the tested build has SSH/NETCONF/gNMI disabled). Re-verified on a real arcos:8.2.1A.P2 container: netlab up deploys via the mapped script and OSPF reaches FULL / iBGP ESTABLISHED.

2. Platform tests. Removed tests/integration/platform/arcos/ entirely — the device is exercised by the regular integration tests.

3. Jinja2 indentation. Applied the readable nested-indentation convention (tag keyword indented with the block) across the ArcOS templates.

Rebased onto current dev.

@ipspace ipspace left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of nits and a few major showstoppers:

A) Changes to documentation (file placement, caveats)
B) IPv4/IPv6 feature parity
C) Correct configuration of IPv4-only, dual-stack and IPv6-only environments. I'm pretty sure we're not testing every potential error, so incorrect configurations could pass the integration tests just because we never thought that someone could do X.
D) Global/VRF feature parity. We expect people to use same/similar configuration for global and VRF routing protocols, and thus do not check every detail like support for areas or OSPF cost in VRF OSPF instances.
E) OSPF areas

Also, while I cannot test stuff (and wouldn't even bother with a vendor that does not have a way to download images for non-customers), I would strongly recommend you run the full set of integration tests. I'm sure some of them (like OSPF multi-area test) would fail.

Comment thread docs/labs/arcos.md
@@ -0,0 +1,54 @@
# Installing Arrcus ArcOS

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file should be in docs/labs. The "docs/install" directory is for netlab installation, the "docs/labs" directory is for providers and devices.

Comment thread docs/release/26.07.md Outdated
* The [**WireGuard tunnel** plugin](plugin-tunnel-wireguard) supports WireGuard tunnels on FRR.
* The [**bgp.session** plugin](plugin-bgp-session) and the [OSPF module](module-ospf) support graceful restart on Arista EOS, BIRD, FortiOS, and FRR
* The [**bgp.policy** plugin](plugin-bgp-policy) supports the **bgp.role** attribute on FRR and BIRD.
* [Arrcus ArcOS](https://arrcus.com/) is now supported (best-effort) as a native containerlab node, with configuration deployment and validation over a `docker exec` + `confd_cli` path.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be in 26.08. 26.07 is already out.

Comment thread docs/caveats.md Outdated
## Arrcus ArcOS

* ArcOS ships a commercial NOS image; there is no public Vagrant/Containerlab box. ArcOS runs as a **native** containerlab node (`kind: arrcus_arcos`) -- no vrnetlab packaging. Provide your own container image and tag it as in `netsim/devices/arcos.yml` (`clab.image`), or override `clab.image` in your topology.
* The tested container image boots with **SSH, NETCONF, and gNMI all disabled** (its baked-in startup config fails to load a built-in prefix-set). *netlab* bootstraps the node with the native `netlab_start_exec` group_var (containerlab's post-start `exec:`), which enables `ssh-server`, changes the factory-default admin-user password (ArcOS refuses to enable interfaces until this is done), and creates an AAA user. No custom deploy code and no SSH is used to run it.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just point to the labs/arcos.md document instead of repeating the points already made there.

Comment thread docs/caveats.md Outdated
* The tested container image boots with **SSH, NETCONF, and gNMI all disabled** (its baked-in startup config fails to load a built-in prefix-set). *netlab* bootstraps the node with the native `netlab_start_exec` group_var (containerlab's post-start `exec:`), which enables `ssh-server`, changes the factory-default admin-user password (ArcOS refuses to enable interfaces until this is done), and creates an AAA user. No custom deploy code and no SSH is used to run it.
* Configuration is deployed with netlab's native containerlab **"sh" config mode**: each module is rendered into `/config/netlab/NN-<module>.sh` and executed via `docker exec`, loading config through `confd_cli` (`load merge` / `commit`) via the mapped wrapper `netsim/templates/provider/clab/arcos/netlab-config.j2` -- the same mapped-script pattern Juniper cRPD uses, with no Ansible in the deploy path. Validation/collect use `ansible_connection: docker` (`docker exec` running `confd_cli`). ArcOS ships the official `arrcus.arcos` `network_cli` collection (the project-recommended model), but every published version (2.0.13-2.0.18) hangs against this image: its cliconf/terminal plugins issue a `config terminal` / `arcos_cli` mode-switch that this build's `confd_cli` rejects, so it is not used.
* `netlab validate` reads ArcOS state over the **same** docker-exec path, running `show <path> | display json | confd_cli` and parsing the OpenConfig JSON (`netsim/validate/**/arcos.py`). This uses netlab's standard device-side show-command validation -- **no** `ansible` validation action, and no SSH/NETCONF/gNMI.
* **MTU:** the `interface <if> mtu <n>` leaf is not a pass-through -- ArcOS programs the real kernel netdev MTU (and the value it advertises in OSPF DBD packets) to `<n> - 26` on every commit. `netsim/ansible/templates/initial/arcos.j2` therefore sets the leaf to `l.mtu + 26` so ArcOS lands on netlab's intended MTU; this alone makes an ArcOS<->FRR OSPF adjacency converge, with no containerlab/core change.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would just say that initial device configuration sets ArcOS to lab topology layer-3 MTU (interface.mtu) + 26.

Comment thread docs/caveats.md Outdated
* Configuration is deployed with netlab's native containerlab **"sh" config mode**: each module is rendered into `/config/netlab/NN-<module>.sh` and executed via `docker exec`, loading config through `confd_cli` (`load merge` / `commit`) via the mapped wrapper `netsim/templates/provider/clab/arcos/netlab-config.j2` -- the same mapped-script pattern Juniper cRPD uses, with no Ansible in the deploy path. Validation/collect use `ansible_connection: docker` (`docker exec` running `confd_cli`). ArcOS ships the official `arrcus.arcos` `network_cli` collection (the project-recommended model), but every published version (2.0.13-2.0.18) hangs against this image: its cliconf/terminal plugins issue a `config terminal` / `arcos_cli` mode-switch that this build's `confd_cli` rejects, so it is not used.
* `netlab validate` reads ArcOS state over the **same** docker-exec path, running `show <path> | display json | confd_cli` and parsing the OpenConfig JSON (`netsim/validate/**/arcos.py`). This uses netlab's standard device-side show-command validation -- **no** `ansible` validation action, and no SSH/NETCONF/gNMI.
* **MTU:** the `interface <if> mtu <n>` leaf is not a pass-through -- ArcOS programs the real kernel netdev MTU (and the value it advertises in OSPF DBD packets) to `<n> - 26` on every commit. `netsim/ansible/templates/initial/arcos.j2` therefore sets the leaf to `l.mtu + 26` so ArcOS lands on netlab's intended MTU; this alone makes an ArcOS<->FRR OSPF adjacency converge, with no containerlab/core change.
* **Static routing validation** reads the committed route from the running-config JSON rather than an operational RIB: this virtual image does not populate an OpenConfig STATIC operational-state/RIB tree. A present, complete static-route entry proves netlab rendered it and the device accepted and committed it; kernel FIB install and forwarding were verified separately.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't go into so many details. I would just say something along the lines of "netlab configures static routes and they are used for traffic forwarding, but the tested ArcOS image does not report them in OpenConfig RIB tree"

{% endif %}
{% if routing.policy|default({}) %}
{% for p_name, p_entries in routing.policy.items() %}
routing-policy policy-definition {{ p_name }}-ipv4

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be IPv4-only. We can do both protocols.

{% endfor %}
{% endif %}
{% if routing.static|default([]) %}
network-instance default protocol STATIC default

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yet again, seems to be the exact same configs for IPv4 and IPv6, so maybe use an inner loop.

Comment thread netsim/ansible/templates/vlan/arcos.j2 Outdated
{% if l.type|default("") == "svi" %}
interface {{ l.ifname }} enabled true
interface {{ l.ifname }} autostate false
{% if l.ipv4 is defined and l.ipv4 %}

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about IPv6 over VLANs? Also: MTU?

{% set nt = l.ospf.network_type|default('point-to-point' if l.type|default('') == 'p2p' else '') %}
{% if nt == 'point-to-point' %}
network-type POINT_TO_POINT_NETWORK
{% endif %}

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about cost, passive interfaces, and the like? We usually define macros in OSPF/BGP/IS-IS templates and then reuse them in VRFs

Comment thread netsim/ansible/templates/ospf/arcos.j2 Outdated
network-instance default protocol OSPF3 p{{ pid }}
global router-id {{ ospf.router_id }}
global log-adjacency-changes LOG_ADJ_ENABLE_DETAILED
area {{ area3 }}

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see where you define other areas

roc-ops and others added 22 commits July 24, 2026 11:17
Address the CHANGES_REQUESTED review (IPv4/IPv6 feature parity; correct config
for IPv4-only, dual-stack and IPv6-only topologies; global vs VRF parity; OSPF
areas; docs). Verified on a real arcos:8.2.1A.P2 container.

- initial: one loop over netlab_interfaces configures every L3 interface
  (system loopback, additional loopbacks, LAG aggregates, routed ports) with
  IPv4 and IPv6 rendered side by side at the interface's actual prefix length
  -- no hardcoded /32 or /128, no hand-built loopback0 name. LAG aggregates now
  get IPv6 addresses too.
- isis: every IPv4-AF enable (global/loopback/interface) is gated on
  isis.af.ipv4, so an IPv6-only box emits no IPv4 IS-IS config; redistribution
  renders per address family (STATIC ISIS IPV4 + IPV6). IPv6 IS-IS is
  single-topology -- this build has no multi-topology knob -- so cross-vendor
  IPv6 IS-IS does not interoperate; documented as a caveat and IS-IS MT set to
  unsupported in platforms.md.
- routing: routing-policy definitions and static routes render per address
  family (inner AF loop) rather than IPv4-only.
- bgp: redistribution renders per address family and now actually commits --
  it needs a #COMMIT transaction split (the BGP instance must exist first) plus
  an explicit accept import-policy (ArcOS rejects default-import-policy
  ACCEPT_ROUTE on the table-connection, same as IS-IS).
- ospf: multi-area support -- one area block per ospf.areas entry, each
  interface filed under its own ospf.area (ArcOS accepts multiple area blocks
  per instance); verified an ABR forms adjacencies in several areas at once.
- vrf: per-VRF OSPF and IS-IS kept consistent with the global-table templates
  (per-interface OSPF metric/bfd/priority/timers + multi-area; dual-AF IS-IS).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015pc2sU3nSrzJNWBi1XDmEw
ArcOS accepts only letters, digits and hyphens in a hostname, so a netlab node
name containing an underscore (dut_a, dut_spine, ... as used by the multi-DUT
integration topologies) aborts the whole initial commit with

  Aborted: system hostname: Valid characters for hostnames are letters,
  digits and the hyphen (-).

Replace _ with - the way the eos/frr/sonic/vyos initial templates already do.
The Jinja comment that documents the native-VLAN handling used whitespace-control
markers, and the leading {#- ate the newline that terminates the preceding
"switched-vlan interface-mode TRUNK" line. The next config line was appended to
it, and ConfD rejected the result:

  Error: on line 1: interface swp1 ethernet switched-vlan interface-mode
  TRUNKinterface swp1 ethernet switched-vlan trunk-vlans [ 502 ]

Drop the whitespace-control markers -- the comment sits on its own lines and
trim_blocks already removes them.
A routed VLAN in a trunk (vlan.mode route) never got any interface configuration:
the initial template skipped every virtual_interface, so the OSPF template was the
only one referencing the subinterface and the commit aborted with

  Aborted: illegal reference 'network-instance default protocol OSPF p1 area
  0.0.0.0 interface swp1.1 subinterface'

An ArcOS routed VLAN subinterface is "subinterface <id>" under the parent port,
and the subinterface ID is the 802.1Q tag: creating "interface swp1 subinterface
701" makes the device report back "vlan vlan-id 701", and the resulting interface
is named swp1.701. So name subinterfaces {ifname}.{vlan.access_id} (as eos, frr,
srlinux and the Junos devices do) and render them from the initial template's L3
interface loop, which keeps their IPv4 and IPv6 addressing in sync with every
other interface type. An untagged (native) routed VLAN keeps its addressing on the
parent port's subinterface 0 -- what ArcOS uses for untagged traffic -- and is
therefore already covered by the same loop.
ArcOS follows the OpenConfig model in which the switched-vlan container hangs off
'ethernet' on a physical port but off 'aggregation' on a LAG bond, so the VLAN
template's hard-coded 'ethernet' was rejected on every bond:

  Error: on line 1: interface bond1 ethernet switched-vlan interface-mode TRUNK

(confirmed on the device: the same line under 'aggregation' commits and shows up
as "aggregation switched-vlan interface-mode TRUNK"). Pick the container from the
interface type for the access, native and trunk-vlans lines alike.
The SVI branch of the VLAN template configured an IPv4 address and nothing else,
so an IPv6-only or dual-stack IRB VLAN got an SVI with no subinterface at all.
OSPFv3 then could not reference it:

  Aborted: illegal reference 'network-instance default protocol OSPF3 p1 area 0
  interface vlan500 subinterface'

Render IPv4 and IPv6 side by side from the same interface data, the way
templates/initial/arcos.j2 already does for every other L3 interface (and use
ansible.utils.ipaddr consistently with the rest of the device's templates).

Also set the interface MTU leaf to l.mtu + 26 -- the same device-side
compensation initial/arcos.j2 applies, because ArcOS programs the netdev and the
MTU it advertises in OSPF DBD packets as (leaf - 26). Without it an SVI silently
inherits the widest member port's MTU, which left an OSPF adjacency over a
reduced-MTU VLAN stuck in ExStart.

Verified with ospf/ospfv3/07-vlan-mtu and ospf/ospfv2/06-vlan-mtu, both of which
now deploy and pass validation.
Two VRFs may legitimately use the same address on their loopbacks, but the L3VRF
network-instances and the interface bindings were only rendered by the vrf module,
which runs after the initial configuration. Both loopbacks were therefore still in
the default network-instance when their addresses were committed, and ArcOS
refused the second one:

  Aborted: 'interface loopback1 subinterface 0 ipv6 address 2001:db8:c001:cafe::1':
  IPv6 subnet overlaps with 'interface loopback2 subinterface 0 ipv6 address
  2001:db8:c001:cafe::1 prefix-length 64'

Create the L3VRF instances and bind the interfaces from the initial template, the
way the eos/ios/junos initial templates do, so the binding is part of the same
transaction as the addressing. templates/vrf/arcos.j2 keeps the routing protocols
inside each VRF and keeps its binding loop, which is a no-op for what the initial
template already bound and still picks up the interfaces created by the modules in
between -- an SVI cannot be bound before the VLAN template creates it ("Interface
vlanN subinterface 0 does not exist").

vrf/04-multi-vrf-loopback-ipv6 and vrf/22-multi-vrf-bgp-ipv6 now deploy.
netsim/devices/arcos.yml declares routing.policy match/set capabilities, so the
bgp.policy plugin accepts locpref/MED/community attributes on ArcOS BGP
neighbours and then refused to configure the lab because the device had no
per-device policy template:

  UserWarning in configs: Cannot find bgp.policy configuration template for
  dut/device arcos

Apply the plugin-generated policies to the BGP neighbours with the OpenConfig
per-AF apply-policy leaf-lists, picking the per-AF policy-definition names that
templates/routing/arcos.j2 emits. Verified on the device with
bgp.policy/21-locpref: the peer sees the DUT's prefixes with local preference 17
and 42 on IPv4 and IPv6 alike.

Only the default network-instance is configured -- templates/bgp/arcos.j2 does not
create a BGP instance inside an L3VRF (this build rejects both 'global as' and
'global afi-safi' under a VRF BGP instance), so there is no VRF neighbour to
attach a policy to. bgp.weight, bgp.bandwidth, bgp.role and AS-path prepending
have no ArcOS equivalent and are not declared in the device settings, so the
plugin rejects them before reaching this template.
MTU is a per-port property on ArcOS: there is no per-subinterface mtu leaf, and an
'mtu' written in subinterface context silently lands on the parent interface
(verified on the device -- 'interface swp1 / subinterface 701 / mtu 1600' shows up
as 'interface swp1 mtu 1600'). A routed VLAN subinterface must therefore not
render one: it would overwrite the parent port's MTU, and on a trunk carrying
several routed VLANs the last one would win. The parent port still gets its own
MTU from its own pass through the same loop, and the subinterface inherits it.

No test in the suite sets an MTU on a routed VLAN link, so nothing changes today;
vlan/51-vlan-routed-trunk still deploys and passes validation.
The VRF-binding loop added in "put VRF interfaces into their VRF before
addressing them" had no filter, while the interface-configuration loop above
it skips SVIs and other virtual interfaces. It therefore emitted a binding for
an interface that does not exist yet, and ArcOS aborts the whole initial commit
on it:

  Aborted: 'network-instance customer interface vlan1000': Interface vlan1000

which broke evpn/03-vxlan-symmetric-irb, vxlan/04-vxlan-irb-ospf and
vxlan/05-vxlan-router-stick. templates/vrf/arcos.j2 already re-binds the
interfaces created by the modules in between, so those interfaces just have to
be left to it.

Collect the interfaces the configuration loop handles in a namespace list and
bind from that list instead of repeating the loop's filter, so the two can no
longer drift apart.
templates/gateway/arcos.j2 renders VRRP only -- its own header says the anycast
form is untested on this image -- so features.gateway.protocol must not list it.
With the declaration in place netlab accepted an anycast topology and rendered an
empty gateway configuration; it now refuses it at transform time
(gateway/01-anycast) or drops the anycast gateway with a warning and falls back
to the real SVI address (evpn/03-vxlan-symmetric-irb).
The VRRP template only ever looked at the interface IPv4 address, so a
dual-stack or IPv6-only first-hop-redundancy group got no IPv6 virtual address
at all. ArcOS hangs the VRRP group off the interface address of the matching
address family and exposes the identical leaf set under both (verified on the
device: virtual-address, priority, advertisement-interval and accept-mode all
commit under ipv6 address), so both families are now rendered from one loop and
an interface may be IPv4-only, IPv6-only or dual-stack.

Also record why gateway.vrrp.preempt is not rendered: this image has no preempt
leaf under vrrp-group in either address family.
features.bgp.activate_af was declared but the template ignored n.activate and
activated whichever address family the neighbor happened to have an address for,
so selective activation (bgp/20-dual-stack-activate) could not work.

A dual-stack peer is two single-AF sessions on ArcOS, so the flag applies per
session: the IPv4-transport session carries the IPv4 AF and the IPv6-transport
session the IPv6 AF -- the same address the eos and frr templates key their
neighbor activation on. A session whose address family is not activated is not
configured, because ArcOS cannot bring up a neighbor with no address family.
VRRP renders and commits for both address families, but this container image
ships no VRRP process: an FRR neighbour with a lower priority stays Master and
counts zero advertisements received from ArcOS, and "show interface <if>" fails
with "application communication failure" as soon as a VRRP group is configured
on it. gateway/02-vrrp and gateway/03-vrrp-vlan therefore deploy but do not
converge, in the same way SR-MPLS, MPLS/LDP and SRv6 already commit without an
exercised datapath. Also record that gateway.vrrp.preempt has no leaf on this
image, and add anycast gateways to the list of deliberately undeclared features.
The address-family, network and neighbor rendering moves into
templates/bgp/arcos.macro.j2 so the per-VRF BGP instances can reuse it
instead of reimplementing the same lines (global vs VRF parity). The AS
and the router-id stay with the caller: a VRF BGP instance has no
"global as" at all, and it takes its router-id from a different place in
the data model.

The macros take the dictionary holding the BGP instance address-family
flags and iterate over [ ipv4, ipv6 ] instead of assuming IPv4 is always
there, so an IPv6-only topology no longer gets a stray IPV4_UNICAST
address family (and a stray STATIC->BGP IPv4 table-connection).

The neighbor macro also carries the bgp.activate gating the
default-instance template applied before the move: a dual-stack peer is
two single-AF sessions on ArcOS, so n.activate[af] is applied per
session, and a session whose only address family is not activated is not
configured at all (ArcOS cannot bring up a neighbor with no address
family). The gate now covers the VRF path too, where it is a no-op:
netsim/modules/bgp.py sets activate[af] unconditionally on every VRF
neighbor carrying that AF because the flag is not user-controllable
inside a VRF.

The rendered configuration is otherwise unchanged, and that is structural
rather than lucky: netsim/modules/bgp.py sets node.bgp.<af> whenever any
neighbor or any advertised prefix carries that AF, only ever to True, and
it is not a user-settable attribute -- so for the default routing table
the new address-family gate is a tautology.
netsim/devices/arcos.yml has declared vrf.bgp since the device was added,
but no template ever rendered a BGP instance inside a VRF, so every PE-CE
VRF BGP session was missing.

Render a BGP protocol instance in each L3VRF network-instance, reusing the
macros from templates/bgp/arcos.macro.j2 so the VRF instance gets the same
router-id, address families, networks and (IPv4 and IPv6) neighbors as the
default-instance BGP.

ArcOS specifics verified on the device:

* the AS number is not part of the per-VRF model (there is no "global as"
  inside an L3VRF); the VRF instance inherits the AS from the default
  instance, which is always configured when a VRF has BGP neighbours;
* "global afi-safi <AF>_UNICAST" has to precede a neighbor activating that
  AF, but the instance, its address families and its neighbours all commit
  in a single transaction -- no #COMMIT split is needed;
* the protocol instance tag is the VRF name, matching
  templates/evpn/arcos.j2 so the EVPN L3VNI configuration lands on the same
  instance.

Route-target import/export leaking is still not rendered: it needs
L3VPN_IPV4_UNICAST on the default-instance BGP, which would change every
global-table session, and it is not a declared feature.
VRF route-target import/export leaking is still not supported, but the
stated reason was wrong: this image does have a route-distinguisher and a
route-target model (an L3VRF BGP instance accepts "global
route-distinguisher", and route targets go under "global afi-safi
<AF>_UNICAST rt-afi-safi L3VPN_IPV4_UNICAST route-target <rt> both" once
L3VPN_IPV4_UNICAST is enabled on the default-instance BGP). Only the
"no rd/RT/L3VPN model" explanation is removed; the caveat stays.
Now that templates/vrf/arcos.j2 renders a BGP instance inside an L3VRF, a
bgp.policy attached to a VRF neighbor is reachable -- and was being dropped
without a word: the template only ever looked at bgp.neighbors, never at
vrfs.<name>.bgp.neighbors, so the policy silently did not reach the device
even though templates/routing/arcos.j2 had already rendered the
policy-definition it referenced.

Render the per-VRF apply-policy too, through a macro shared with the
default-instance path. The plugin's configuration is deployed after the vrf
module, so the instance a VRF neighbor lives in already exists.

The comment claiming the platform "rejects global as and global afi-safi
under a VRF BGP instance" was also wrong: only global as is rejected, and
this is the same wrong reason that was just removed from docs/caveats.md.

The default-instance output is byte-identical.
…rted

It was listed under "Genuinely unsupported, not merely untested", which the
evidence contradicts: the configuration model is there. An L3VRF BGP
instance accepts "global route-distinguisher" (every VRF BGP instance
netlab now renders carries one), and route targets go under "global afi-safi
<AF>_UNICAST rt-afi-safi L3VPN_IPV4_UNICAST route-target <rt> both".

What blocks it is that the route-target leaf is rejected until
L3VPN_IPV4_UNICAST is enabled on the DEFAULT-instance BGP, which changes the
capabilities advertised on every global-table session. That was out of scope
for the VRF BGP work, and the leaking itself has never been tested, so it
stays unrendered and undeclared -- but as "not implemented", not as a
platform limitation.
… templates

Both templates write "global route-distinguisher" on the same
"network-instance <vrf> protocol BGP <vrf>" instance, and evpn is deployed
after vrf (netsim/modules/evpn.yml config_after), so evpn won -- meaning a
VRF that had both an L3VNI and BGP neighbors would silently commit
<router-id>:<transit_vni> instead of netlab's vrfs.<name>.rd.

Use vdata.rd in the EVPN template as well, the way templates/vrf/eos.bgp.j2
renders "rd {{ vdata.rd }}" whether or not EVPN is in play. The two writers
now emit an identical leaf, so the committed RD matches the data model
regardless of ordering.

No test currently renders both blocks on one instance, so this changes
exactly one rendered line today: evpn_03-vxlan-symmetric-irb goes from
10.8.0.5:5042 to 65000:1.
…vdata.af

The header of templates/bgp/arcos.j2 claimed the AS and the router-id moved
into templates/bgp/arcos.macro.j2. They did not, and cannot: a VRF BGP
instance has no "global as" at all and takes its router-id from a different
place in the data model, so both stay with the caller. The macro file's own
header already said this correctly.

Also pass vdata.af through |default({}) in templates/vrf/arcos.j2, matching
templates/vrf/eos.bgp.j2. netlab's Jinja2 environment uses a chainable
undefined, so a VRF without an "af" dictionary would not raise -- it would
render an instance with no address families and no neighbors. Unreachable
today; the guard costs nothing.
The evpn template iterated vlans.items() and vrfs.items() unguarded, so any
EVPN topology whose node defines neither collection aborted at transform with

  Error rendering template evpn for node dut/device arcos ... items is undefined

That broke eight integration tests outright -- evpn/01, 02, 05, 10, 11, 12, 21
and 22 -- on every branch, because a spine or a route reflector carries the EVPN
address family without owning a single VLAN or VRF.

Guard both loops with |default({}). Note the parentheses: vlans|default({}).items()
parses as vlans|default(...) applied to .items() and is a Jinja syntax error, so
the filter has to be wrapped -- (vlans|default({})).items().
roc-ops added 15 commits July 27, 2026 07:47
…tate mechanism

The substance of the previous caveat was right but two details were wrong, and
an independent five-lab tiebreak established both.

* The displacement is tied to the target's neighbour entry being REACHABLE, not
  to "the target sends anything to the device". The correlation is one-to-one,
  per target and per address family: on a settled lab the only discard still
  installed was the one whose target had no neighbour entry, while every target
  showing REACHABLE had none. The RIB keeps the discard as a candidate (origin
  STATIC, interface null0) and reports best-protocol ADJACENCY for the same /32.
  It is also sticky: shutting the target's interface for 90 s left the neighbour
  REACHABLE with no blackhole, and a hand-deleted neighbour entry is reinstated
  immediately.

* routing/25-static-discard cannot be made reliably green on this platform by
  any wait value, which the previous wording did not say. For roughly the first
  three minutes after the DUT starts the STATIC daemon has installed nothing, so
  the discards do not exist; then all four appear for some 16-21 seconds while
  the targets are still STALE; then the targets go REACHABLE and the blackholes
  leave the FIB for good. Both ends of that timeline fail and only the narrow
  middle passes, so there is no wait value that can target it -- five fresh labs
  all reported 7/11.

The docs/caveats.md bullet is the tiebreak's text verbatim rather than a
paraphrase, so the measured timeline is recorded exactly; the support.caveats
entry in netsim/devices/arcos.yml carries the same two corrections.

Documentation only. features.routing.static.discard stays declared: discard
works correctly for a prefix that is not directly connected.
…rything

An ArcOS prefix-set is a flat member list -- the accept/reject decision lives on
the policy statement that references it, not on the members. A netlab prefix
list has an action per entry with first-match-wins semantics, where a 'deny'
entry means "this prefix does NOT match the list".

The template ignored the action and put every entry in one set, so a policy that
rejects on a match of the list rejected the deny entries too. routing/10-match-
prefix has been deploying that way:

  routing-policy defined-sets prefix-set orig_1
   prefix 172.42.43.0/24 exact     <- a DENY entry: must NOT match, so must not be a member
   prefix 172.42.42.0/24 exact

Measured on the device before the fix: the DUT's BGP RIB holds neither
172.42.43.0/24 (which the topology permits) nor 172.42.42.0/24 (which it
denies) -- everything the list mentions was filtered. Silent: the configuration
commits and reads plausibly.

Render the PERMIT entries only. The deny entries are then simply absent from the
set, do not match, and fall through to the policy's later statements -- which is
exactly netlab's semantics as long as no deny entry shadows a permit entry below
it (refused by a quirk in the following commit).

A list with no permit entries in an address family matches nothing there. That
needs care: emitting an empty prefix-set would leave the statement with no
condition at all, i.e. matching every prefix and applying the opposite outcome.
So the set is not emitted and the statement referencing it is skipped -- visible
in the IPv6 half of routing/10, where in_x1-ipv6 now carries only its catch-all.

routing/10-match-prefix 20/24 -> SUCCESS 24/24, verified on the wire: the test
asserts both that permitted prefixes reach x3/vx3 and that denied ones are
absent. It is the only integration test using action: deny in a prefix list
(routing/11 uses one in an as-path list, which ArcOS declines outright; 12 and
13 use community lists). Render sweep of all 261 topologies for -d arcos: this
is the only configuration that changes, and every changed line is a removal.
Rendering the permit entries only (previous commit) is faithful as long as no
deny entry overlaps a permit entry that comes after it. When they overlap, order
is the only thing separating them, and a flat ArcOS prefix-set cannot carry
order:

  - ipv4: 10.1.0.0/16
    action: deny
  - ipv4: 10.0.0.0/8            # 10.1.x would still match this member

10.1.0.0/16 should not match the list, but with only 10.0.0.0/8 in the set it
does. The result would be a route-map matching the wrong routes -- silent, and
visible only as traffic taking the wrong path -- so refuse it up front, in the
same form as the existing evpn_static_flooding and community_list_shape quirks.

The overlap test is on the networks themselves: if the deny and permit prefixes
do not overlap, no address can match both whatever their min/max lengths are, so
comparing networks is sufficient and errs toward accepting.

Verified both directions: the quirk fires on the topology above with the two
prefixes named in the message, and does not fire on any of the 261 integration
topologies rendered for -d arcos -- including routing/10-match-prefix, whose
deny and permit entries do not overlap.
… measured

The caveat said the ABR emits a /0 network mask for a /32 loopback and that
"IPv4 and IPv6 (OSPFv3) behave the same". Reading the neighbours' link-state
databases while triaging four near-miss tests shows the defect family is wider
than that, and that the two address families fail differently.

* OSPFv2 -- confirmed in the LSDB rather than inferred from a missing route. The
  ABR's own summary LSA, read from an area-0 peer:
      Link State ID: 10.8.0.1   Advertising Router: 10.8.0.1   Network Mask: /0
  while the /30 transit prefixes in the same LSA set carry Network Mask: /30.
  So it is specific to host routes, which the old text asserted but had not
  shown.
* OSPFv3 -- NOT the same. The loopback is advertised with the interface prefix
  length instead of /128: a peer installs 2001:db8:1:1::/64 where
  2001:db8:1:1::1/128 is expected. The prefix is present, just not as a host
  route -- a quieter failure than the IPv4 form, and one the old wording would
  have led a reader to look for in the wrong place.
* In a VRF -- not mentioned at all before. A per-VRF OSPF instance acting as an
  ABR shows the identical Network Mask: /0 for the VRF loopback, alongside a
  correctly summarised /30, read from the VRF peer's database.

Between them these explain ospf/ospfv2/02-areas, ospf/ospfv3/06-lb-prefix,
vrf/21-multi-vrf-ospfv3 and two of the three failing checks in
vrf/11-multi-vrf-ospf. Nothing is rendered for any of it -- this is an
LSA-origination defect, not a configuration gap -- so this is documentation
only, with no rendering change.
# Conflicts:
#	netsim/ansible/templates/routing/arcos.j2
#	netsim/devices/arcos.py
# Conflicts:
#	docs/caveats.md
#	netsim/devices/arcos.yml
Left to itself ArcOS derives the IP-VRF route target as <AS>:<L3VNI> -- 65000:5042
for a transit VNI of 5042 -- while netlab, and every other netlab EVPN device,
derives it as <AS>:<vrf-id> from vrfs.<name>.import/export, i.e. 65000:1. The two
never match, so neither side imports the other's type-5 route and symmetric IRB
does nothing at all.

Measured on the container image (arcos:8.2.1A.P2), evpn/03-vxlan-symmetric-irb:

  before   DUT  evpn-v4-import-route-targets 65000:5042
           FRR  L3 VNI 5042  Import RT 65000:1  Export RT 65000:1
           DUT customer VRF loc-rib: only its own two prefixes

  after    DUT  evpn-v4-import-route-targets 65000:1
           DUT customer VRF loc-rib: 172.18.2.0/24 via 10.8.0.6  <- the peer's subnet
           h1 -> h4 ping starts working (a check that was failing)

Two placement details, both verified on the device and easy to get wrong:

* the RT does not live where the MAC-VRF one does. For an L2VLAN instance it is
  "protocol BGP <ni> route-target <rt> both"; an L3VRF has no such node and the
  value sits under the unicast AFI, in an rt-afi-safi L2VPN_EVPN container, with
  a mandatory direction keyword;
* it is per address family. Setting it only under IPV4_UNICAST left the IPv6
  type-5 routes still carrying the auto-derived 65000:5042, so both families are
  rendered from the same netlab lists.

No feature declaration changes here: the egress half of symmetric IRB still does
not work (see the caveat correction that accompanies this), so nothing new is
claimed. This removes a wrong route target from the rendered configuration and
makes the import direction behave.
…advertised

netsim/devices/arcos.yml declared under "genuinely unsupported" that EVPN
symmetric IRB / L3VNI does not work because "route-type-5 is never originated".
Our own hardware validation had already overturned that on S8.5.1A, and it is
wrong on the container image too. Measured on evpn/03-vxlan-symmetric-irb with
arcos:8.2.1A.P2:

  show overlay vni-all 5042
    vni-state vni-run-state UP
    vni-state vni-type      ROUTED
    vni-state ip-or-mac-vrf customer

  default instance, L2VPN_EVPN loc-rib
    65000:1:[5][0][24][172.18.0.0]     ext-community [ route-target:...
    65000:1:[5][0][24][172.18.1.0]       encap-type:VXLAN router-mac:02:42:c0:a8:8d:6a ]
    65000:1:[5][0][64][2001:db8:172::]
    65000:1:[5][0][64][2001:db8:172:1::]

Four well-formed type-5 routes, one per connected subnet in the VRF. Import
works as well once the route target is written explicitly (the accompanying
template fix): the peer's subnet appears in the VRF loc-rib and a host behind
this PE starts reaching a host behind the peer.

The real gap is the egress half. The locally originated type-5 routes carry
path-types [ BEST_PATH EXPORTED_PATH ] and no rib-out entry -- they are never
advertised to the EVPN peer, so the peer never learns this PE's subnets.
"rt-afi-safi L2VPN_EVPN evpn-route-options suppress-rt5-routes false" commits
and does not change it.

Symmetric IRB stays undeclared -- nothing new is claimed -- but the recorded
reason now matches what the device does. Documentation only.
templates/bgp/arcos.j2 renders a neighbor only for an address family the
neighbor activates (bgp.activate, features.bgp.activate_af), because ArcOS
cannot bring up a session that has no address family. An EVPN-only session has
activate.ipv4 == activate.ipv6 == false -- its address family is L2VPN_EVPN --
so the whole neighbor block was skipped there, and templates/evpn/arcos.j2
created the neighbor with no AS at all.

The device accepted it and reported:

  neighbor 10.8.0.3
   state peer-as       0
   state session-state INCOMPLETE

with the peer sitting in Active forever (evpn/12-vxlan-ibgp-ebgp). Emitting
peer-as alongside the L2VPN_EVPN activation fixes it; it is idempotent when the
BGP template also rendered the neighbor, being the same leaf with the same value.

Verified on arcos:8.2.1A.P2 -- the EVPN session to the other ArcOS node goes
from INCOMPLETE to:

  neighbor 10.8.0.3
   state peer-as       65000
   state session-state ESTABLISHED
netlab's replace_global_as (default true) means the local AS REPLACES the global
one in the AS path rather than being prepended to it -- "no-prepend replace-as"
on FRR, and two identically named leaves under ebgp-local-as here. The template
rendered only "local-as", so the global AS stayed in the path.

That is invisible until a peer's own AS equals the global one, which is exactly
what an IBGP-over-EBGP design does. In evpn/12-vxlan-ibgp-ebgp the spine
(global AS 65000, local-as 65100) advertised its loopback to an FRR leaf whose
real AS is also 65000; FRR discarded the update as an AS-path loop -- silently,
since a loop is dropped before policy, so it showed "0 accepted" with
"Inbound filtered: 0" -- the leaf never got a route to the spine loopback, and
the EVPN session running over that loopback stayed Active indefinitely.

The other ArcOS leaf accepted the same update, which is why only the FRR peer
was affected and why this looked like an EVPN problem rather than an AS-path one.

Verified on arcos:8.2.1A.P2, adding both leaves on the live spine:

  before  l2: show ip bgp    -> only its own 10.8.0.4/32
          l2: ip route get 10.8.0.5 -> Network unreachable
          l2: EVPN neighbor 10.8.0.5 -> Active, never up

  after   l2: 10.8.0.5/32  10.10.0.5  0 65100 i     <- AS path is the local AS
          l2: EVPN neighbor 10.8.0.5 -> up 00:00:46, 18 messages received

Gated on replace_global_as so a topology that asks for prepend semantics still
gets them.
70f72e9 corrected the claim in netsim/devices/arcos.yml but two copies
survived, which is the same way three copies of the community claim survived on
an earlier branch:

* templates/evpn/arcos.j2 still carried a "KNOWN NOT FUNCTIONAL ... ZERO
  route-type 5 (IP Prefix) routes are ever originated" block;
* docs/caveats.md still listed "a route-type-5 IP-prefix route is never
  originated" under genuinely unsupported.

Both now say what was measured: type-5 routes are originated (one per connected
subnet, RD-qualified, encap-type:VXLAN, per-VTEP router-mac, with the L3VNI
reporting vni-type ROUTED) and can be imported, and the gap is that they are
never advertised to the peer -- no rib-out entry, and suppress-rt5-routes false
does not change it. The template comment also records why the old measurement
said zero: it predates the VRF BGP instance carrying its connected subnets, so
there was nothing to originate.

The same docs/caveats.md sentence also still claimed "redistribution sources
other than static", which the OSPF/IS-IS-into-BGP work made stale. Corrected in
the same edit rather than left in a line being rewritten: redistribution into an
IGP is unsupported, redistribution into BGP works for connected, static, OSPF
and IS-IS.

Documentation only; no feature or template behaviour change.
# Conflicts:
#	docs/caveats.md
#	netsim/devices/arcos.py
#	netsim/devices/arcos.yml
# Conflicts:
#	docs/caveats.md
#	netsim/devices/arcos.yml
…easurement

dbfcb85 narrowed the leaking claim in netsim/devices/arcos.yml and in
templates/vrf/arcos.j2 but never touched docs/caveats.md, so the user-facing
document still carried the pre-measurement text. The two files contradicted each
other: the device file says "measured as unsupported, not merely untested" while
caveats.md said "the leaking itself has never been tested".

caveats.md now says what was measured:

* the RT/L3VPN form -- the one netlab's vrfs.<name>.import/export model maps onto
  -- was measured, not assumed. Its export half completes: with a per-VRF
  label-allocation-mode and an MPLS reserved-label-block using BGP_COMMON, the
  VRF reaches "Export ready" and the per-VRF exported-rib fills with correctly
  targeted, labelled VPN routes. The label block is the step everything waits on,
  so the two intermediate states and the LibLabelAlloc error are recorded for
  whoever retries this;
* the import half never fires for a locally exported route -- default-instance
  L3VPN loc-rib empty, every VRF at "paths-imported 0" while reporting
  "Import ready" with the correct import route targets -- so RT-based leaking
  here needs a remote PE, which a single-device leaking topology has not got;
* ArcOS's OTHER leaking mechanism, which caveats.md did not mention at all:
  "static-route <pfx> next-hop-index N next-network-instance-name <vrf>" works
  and forwards (3/3 between hosts in two VRFs, with a third VRF still isolated)
  but takes host routes only, so it cannot carry netlab's prefix-based lists.
  That is the counter-example a reader would otherwise find and conclude the
  caveat was wrong.

Also corrects a claim this document made that later measurement disproved:
enabling L3VPN_IPV4_UNICAST on the default-instance BGP does NOT change the
capabilities advertised on existing global-table sessions. ArcOS activates
address families per neighbour; on a live bgp/01-ebgp-session lab both sessions
stayed Established with no reset and negotiated no additional address family
after the instance-level AF was added.

Documentation only.
@ipspace
ipspace marked this pull request as draft July 29, 2026 10:36
@ipspace ipspace added the needs changes Use for PRs that need heavy changes before it's ready to be merged. label Jul 29, 2026
@roc-ops

roc-ops commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks Ivan — I took the integration-test recommendation seriously and ran the full set. You were right about multi-area OSPF.

Results: 25 → 77 of 181 tests passing, measured before and after with the same harness and stock addressing, no regressions. Every deploy failure eliminated (24 → 0).

On (E), OSPF areas — your prediction was correct, and here is the mechanism. ospf/ospfv2/02-areas fails. The ABR forms adjacencies in every area and floods correct inter-area Type-3 summaries for transit (e.g. /30) networks, but it summarises a /32 loopback host route with a /0 network mask — read from a neighbour's database, the ABR's own summary LSA for its 10.0.0.1/32 loopback arrives as Link State ID: 10.0.0.1 ... Network Mask: /0, and the neighbour mis-installs it as a 0.0.0.0/0 default. The same LSA set carries the /30 transit prefixes with the correct mask, so it is specific to host routes. OSPFv3 gets it wrong differently — it advertises the loopback with the interface prefix length instead of /128, so the prefix is present but not as a host route. The same thing happens inside a VRF. Net effect: a loopback placed in a non-backbone area is not reachable from other areas; loopbacks in area 0 and all inter-area transit prefixes are unaffected. This is documented in docs/caveats.md rather than silently dropped, and ospf.areas parameters are refused outright rather than half-rendered.

On (B) and (D), IPv4/IPv6 and global/VRF parity — both axes were swept, and gaps were either implemented or declared honestly. Several capabilities that were previously declared unsupported turned out to work, and are now implemented and verified on the device rather than from show running-config:

  • STP — Rapid PVST only (the stp enabled-protocol leaf enumerates NONE | RAPID_PVST, so no 802.1D/RSTP/MSTP). Cross-vendor root election verified against an EOS peer. features.stp.port_type is deliberately not declared, because the flag is a bare boolean and ArcOS has no per-port equivalent of network (bridge assurance is global).
  • BGP large communities — match, set and delete. These were previously rendered as a comment and silently dropped.
  • VRF static routes and discard/blackhole static routes.
  • OSPF conditional default origination — the platform can do it via a table-connection; what it cannot do is always, and cost/type are accepted and then break redistribution, so features.ospf.default stays undeclared and the reason is documented.
  • Routing-policy prefix lists — per-entry actions were being flattened so that a list with a deny entry filtered everything.

Others were removed from the feature list because the platform genuinely cannot do them.

On (A) — the per-platform integration tests are gone, as you asked. The documentation placement I have not fixed yet; it will be in the next push.

On (C) — dual-stack is covered by the suite and passes. IPv6-only and IPv4-only topologies I have not systematically verified, so I would rather say that than claim it.

One process question. Addressing all of this has grown the branch considerably — it now touches 17 template modules. Would you prefer it staged as a series of smaller dependent PRs (device skeleton → IGP → BGP/routing-policy → L2 → overlay → remaining modules), or kept as one? Happy either way; I would rather ask than force-push a restructure onto a review you are part-way through.

snmpstack and others added 2 commits August 1, 2026 17:21
…device

docs/install/ holds general installation documentation (clone, cloud, linux,
ubuntu); per-device pages live in docs/labs/ -- arubacx, asav, ceos and the
rest are all there. The ArcOS page was in the wrong directory.

Also adds the clab.build key pointing at the page, matching the fourteen other
devices that carry one, so the device definition links to its own
documentation.
netlab only sets ospf.af.<family> for the address families a node actually
has, so on an IPv6-only node ospf.af.ipv4 is undefined. The guard read

  {% if ospf.router_id is defined and ospf.af.ipv4|default(true) %}

so the default let it through, the header rendered, and the area loop
underneath correctly produced nothing -- leaving a stranded OSPFv2 process
with a router ID and no areas and no interfaces:

  network-instance default protocol OSPF p1
   global router-id 10.0.0.1
   global log-adjacency-changes LOG_ADJ_ENABLE_DETAILED

The device does not object. Deployed on arcos:8.2.1A.P2, `show running-config
network-instance default` keeps the empty stanza with no error and no warning,
so nothing in the render-commit path flags it, and whether an integration test
notices depends entirely on whether that test happens to check OSPFv2 state.

This was the only |default(true) address-family gate in the ArcOS templates.
The OSPFv3 guard sixteen lines below, all six guards in isis/arcos.j2 and the
OSPF/IS-IS guards in vrf/arcos.j2 already use |default(false).

Verified:

* Render sweep, 47 topologies x 3 address-family modes, base vs branch. 28
  combinations differ and all 28 have the identical shape -- the same three
  lines removed, nothing added anywhere:
      - network-instance default protocol OSPF p1
      -  global router-id 10.0.0.1
      -  global log-adjacency-changes LOG_ADJ_ENABLE_DETAILED
  No IPv4-only render changed at all. Every differing "dual-stack" combination
  is an ospfv3 test, and those are natively IPv6-only
  (tests/integration/ospf/ospfv3/topology-defaults.yml sets ipv4: False on all
  three pools). The genuine dual-stack topology, ospfv3/08-dual-stack, renders
  byte-identically and keeps its full OSPFv2 block.
* On the device: IPv6-only ospfv2/01-network and IPv6-only
  bgp/02-ibgp-ebgp-session now show only `protocol OSPF3 p1` (the second case
  confirms this was never confined to the OSPF tests); IPv4-only
  ospfv2/01-network still shows `protocol OSPF p1` with area 0.0.0.0 and all
  its interfaces; dual-stack ospfv3/08-dual-stack still shows both instances.
* OSPF integration suite (35 tests, ospfv2 + ospfv3), base and branch: 34 of 35
  identical. The one that moved, ospfv3/42-bfd, is unstable on both sides under
  the 3-run protocol -- base PASS/FAIL/FAIL, branch FAIL/PASS/PASS -- so it is
  UNSTABLE rather than a regression, and the base run's PASS was luck.
@ipspace

ipspace commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Results: 25 → 77 of 181 tests passing, measured before and after with the same harness and stock addressing, no regressions. Every deploy failure eliminated (24 → 0).

Does that mean there are zero tests that fail in config or validate phase, or just that the configurations generated through the templates are not invalid?

On (E), OSPF areas — your prediction was correct, and here is the mechanism. ospf/ospfv2/02-areas fails.

I'm guessing this hints at what the above answer might be ;) Anyway, would love to know the exact details.

Both axes were swept, and gaps were either implemented or declared honestly.

There are still several unresolved documentation comments. Even if you feel like the whole thing is ready to be deployed, please address them.

On (C) — dual-stack is covered by the suite and passes. IPv6-only and IPv4-only topologies I have not systematically verified, so I would rather say that than claim it.

There are IPv4-only and IPv6-only BGP/OSPF tests; it's just that in many cases, we don't check whether the tested device has also enabled the other address family.

One process question. Addressing all of this has grown the branch considerably — it now touches 17 template modules. Would you prefer it staged as a series of smaller dependent PRs (device skeleton → IGP → BGP/routing-policy → L2 → overlay → remaining modules), or kept as one?

Keep it as one PR. Once the documentation is polished, I'll merge it (I can't test it anyway 🤷🏻‍♂️)

@roc-ops

roc-ops commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Does that mean there are zero tests that fail in config or validate phase, or just that the configurations generated through the templates are not invalid?

Good question, and the honest answer is "the first for config, definitely not for validate". The full breakdown of the 181:

PASS (deployed and all checks green) 77
FAIL (deployed, one or more validation checks failed) 27
WARN (deployed, checks green, validation emitted a warning) 8
refused at transformation as unsupported 69
failed the config phase 0

So: zero deploy failures — every topology that transforms gets a configuration ArcOS accepts and commits. But 27 tests still fail validation, which is the number I should have led with rather than the 77. It went from 24 config-phase failures to none, and from 45 validation failures to 27.

I'm guessing this hints at what the above answer might be ;) Anyway, would love to know the exact details.

ospf/ospfv2/02-areas is one of the 27, and the mechanism is a genuine ArcOS defect rather than a template problem.

The ABR accepts interfaces in several areas, forms adjacencies in each, and floods correct inter-area Type-3 summaries for transit prefixes. What it gets wrong is a /32 loopback host route: read from a neighbour's database, the ABR's own summary LSA for its 10.0.0.1/32 loopback arrives as

Link State ID: 10.0.0.1 ... Network Mask: /0

and the neighbour installs it as a 0.0.0.0/0 default. The same LSA set carries the /30 transit prefixes with the correct mask, so it is specific to host routes.

OSPFv3 gets it wrong differently — it advertises the loopback with the interface prefix length instead of /128, so the prefix is present but not as a host route. That is the quieter of the two failures. The same thing happens inside a VRF: a per-VRF instance acting as ABR shows the identical Network Mask: /0 for the VRF loopback while summarising that VRF's transit prefixes correctly.

Net effect: a loopback placed in a non-backbone area is unreachable from other areas; loopbacks in area 0 and all inter-area transit prefixes are fine. It is written up in docs/caveats.md, and ospf.areas parameters are refused outright rather than half-rendered.

There are IPv4-only and IPv6-only BGP/OSPF tests; it's just that in many cases, we don't check whether the tested device has also enabled the other address family.

That is exactly the hole, and there was one on our side. Rendering ten topologies across the modules in IPv4-only / dual-stack / IPv6-only turned up an IPv6-only node emitting an empty OSPFv2 instance — router-id and log-adjacency-changes, no area, no interfaces, on a node whose addresses are all IPv6. ArcOS accepts and keeps it without complaint, and no test catches it: the render succeeds, the commit succeeds, and the IPv6-only topologies that do fail, fail on their own IPv4 assumptions and would fail identically with the block absent.

Cause was ospf/arcos.j2 using ospf.af.ipv4|default(true) — the only |default(true) address-family gate in the ArcOS templates, against |default(false) in the OSPFv3 guard, all six IS-IS guards and the VRF guards. Fixed, with the mirror cases verified: IPv4-only still emits its OSPFv2 block, dual-stack still emits both, and a render diff over 47 topologies × 3 modes shows the only change anywhere is the removal of those empty stanzas.

I am now sweeping the rest of the PR for the same class, per your note that it probably applies more widely, and working through the outstanding documentation comments.

roc-ops added 5 commits August 2, 2026 13:06
…o it

Review: "add (build-arcos)= or (labs-arcos)= anchor so you do not have to
hyperlink to a file name" and "add a link to the anchor at the top of
labs/arcos.md" from docs/platforms.md.

Uses (build-arcos), matching (build-frr) / (build-eos) / (build-ceos), and links
the containerlab column of the provider-support table the same way the vEOS row
above it does.
Review: "this should be in 26.08. 26.07 is already out."

docs/release/26.08.md is taken verbatim from origin/dev (this branch predates
it) with the ArcOS entry added under New Functionality and nothing else changed
-- diffing the file against origin/dev shows only that one bullet. The toctree
line in docs/release.md is upstream's own, added here so the new file is
reachable while the branch is still behind dev.

The entry itself is shortened to a single sentence pointing at the two documents
that carry the detail, rather than restating the deployment path.
Review: "I try to keep caveats short", "I would just point to the labs/arcos.md
document instead of repeating the points already made there", and "I would skip
this. Things don't work, they are not implemented, they are not enabled in
features, so you can't use them. Let's move on."

The section was 57 lines and 3973 words -- more than five times the next longest
device in this file (Fortinet FortiOS, 771 words) and eight times Arista EOS.
It is now 33 lines and 343 words.

The cut follows one rule: a caveat earns its place only if _netlab_ accepts the
topology and the result is silently wrong or degraded. Everything a user cannot
reach, because the feature is undeclared or refused during transformation with
its own message, is gone -- OSPF default origination, inter-VRF route leaking,
EVPN symmetric IRB / L3VNI, the BGP link-bandwidth community, per-member actions
in community lists, the STP restrictions, the EVPN-with-static-VXLAN
combination, and the "genuinely unsupported" list. The VRRP file-descriptor
bootstrap is gone too: _netlab_ does it automatically, so it is not something
the reader has to act on.

Installation, first-boot bootstrap, deployment mechanism and validation are no
longer restated here; they are in labs/arcos.md, which this section now links
to. Deployment is described in one clause, as suggested. The VLAN/SVI model note
moved into labs/arcos.md, where it is reference material rather than a caveat.

Kept, and shortened, with the wording from the review where it was given: the
MTU + 26 adjustment, static routes missing from the OpenConfig RIB tree, the
SR-MPLS / MPLS-LDP / SRv6 dataplane not working in the container, the ~70 second
BGP startup delay, the OSPF ABR mis-summarizing loopback host routes, IS-IS IPv6
being single-topology, IPv4 VRRP not interoperating, static VXLAN flooding not
being able to originate BUM traffic, and discard static routes being displaced
by a connected host route.
The same defect as 9234216, in the VRF template. The per-VRF OSPFv2 block was
guarded only by `vdata.ospf is defined`, with no address-family test, while the
per-VRF OSPFv3 block sixteen lines below already had `vdata.ospf.af.ipv6`. netlab
sets vrf.ospf.af.<family> only for the families the VRF has, so an IPv6-only VRF
got a stranded instance -- a router ID and no areas and no interfaces, the inner
`if l.ipv4 is defined` loop correctly producing nothing:

  network-instance customer protocol OSPF p-customer
   global router-id 10.0.0.1

Found by rendering every module ArcOS declares in IPv4-only, dual-stack and
IPv6-only and reading the output, after review noted the address-family question
"probably applies to the rest of the PR as well".

`vdata.ospf.af` read out of the rendered VRF data is `{'ipv4': True}` where the
VRF has IPv4 and `{'ipv6': True}` where it does not, so `|default(false)` can only
suppress a VRF netlab has already said has no IPv4 OSPF.

Verified:

* Render sweep, 88 topologies x 3 address-family modes, base vs branch: 16
  combinations differ, all of them the same two-line removal, zero lines added
  anywhere -- 28 `network-instance <vrf> protocol OSPF p-<vrf>` headers and their
  28 `global router-id` lines, across the red, blue, green, common and customer
  VRFs of nine topologies.
* Three of those combinations are in the v4-only and dual-stack columns, which
  looks wrong and is not: `vrf/21-multi-vrf-ospfv3` and
  `vrf/23-multi-vrf-mixed-ipv6` load `defaults-ipv6-only.yml`, which pins the
  blue VRF to IPv6-only pools whatever the rest of the topology does. Its
  `ospf.af` is `{'ipv6': True}` in every column.
* Mirror case, in the same rendered file: with `vrf/21` in the IPv4-only column,
  `red` (`ospf.af {'ipv4': True}`) keeps `network-instance red protocol OSPF
  p-red` while `blue` keeps only `protocol OSPF3 p-blue`.
* OSPF, VRF and vlan/52 integration tests, 54 in total, base and branch: all 54
  identical.
# Conflicts:
#	docs/caveats.md
#	docs/release/26.08.md
@roc-ops

roc-ops commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Pushed. This is the first time the branch has been updated since your review — you had only ever seen the initial two commits, so there are 90 on top of that, plus a merge of current dev. Summary of what changed and what I verified.

Documentation (the merge condition)

  • docs/caveats.md ArcOS section cut from 3973 words to 343, 57 lines to 33. Nine caveats kept — the ones a user actually hits — and the rest either dropped or moved into docs/labs/arcos.md where the installation and bootstrap detail belongs.
  • Added a (build-arcos) anchor and linked it from the platform table, so the "best-effort / no public image" story is reachable from where a user starts.
  • Release note moved to 26.08.

The address-family sweep

You said it probably applied to the rest of the PR, and you were right — there was a second instance. I rendered every module the device declares in IPv4-only, dual-stack and IPv6-only: 211 topologies, 633 combinations, and read the rendered configuration rather than only scoring tests.

Two shapes were hunted:

"A block emitted for a family the node does not have" — one more defect. vrf/arcos.j2 guarded the per-VRF OSPFv2 block only on vdata.ospf is defined, while the per-VRF OSPFv3 block sixteen lines below already checked vdata.ospf.af.ipv6. An IPv6-only VRF got a stranded network-instance <vrf> protocol OSPF p-<vrf> carrying a router ID and nothing else. Same class as the ospf/arcos.j2 one, missed the first time because no topology in that first pass had an IPv6-only VRF.

Verified the same way as before: 88 topologies × 3 modes, base vs branch, gives 16 differing combinations and all 16 are the identical two-line removal with zero lines added anywhere — 28 stanza headers and their 28 router-id lines, across five VRF names in nine topologies. All 54 OSPF/VRF/VLAN integration tests render identically before and after.

"A block emitted for only one family when both are present" — zero hits. Nowhere in the 633 combinations does a dual-stack render carry fewer constructs of a family than the corresponding single-family render. That is the shape your routing/arcos.j2:28 and isis/arcos.j2:70 comments were pointing at, and it does not occur on the branch.

On your five specific pointers: four of them were already addressed by e58e18decb (IPv4/IPv6 parity) and 010d468f8e (IPv6 addresses + MTU on SVIs), which landed after your review — so those line numbers no longer land on the code they describe. Each is now confirmed by a render: an IPv6-only IS-IS node emits zero IPv4 constructs; dual-stack IS-IS redistribution emits both STATIC ISIS IPV4 and STATIC ISIS IPV6; static routes render both families from one loop; a dual-stack SVI renders the MTU leaf and both addresses. IS-IS is still single-topology, and both the caveat and platforms.md still say so.

One thing I did not change, because I do not think it is mine to decide

routing/arcos.j2 renders every routing policy twice, -ipv4 and -ipv6, unconditionally, so a single-family node carries one dead policy definition. Gating that looks like a two-line fix, and I believe it is unsafe: bgp/07-ebgp-localas sets bgp.originate: 172.42.42.0/24 on a node with no IPv4 interface, so its BGP instance legitimately carries afi-safi IPV4_UNICAST even in the IPv6-only render. Gating the policy on interface address families would drop the -ipv4 policy and leave that reference dangling. The current behaviour is deliberate, documented in the template header, and reference-safe — but it is a real wart, and if you would rather gate it on something else I will take the direction.

Merge of current dev

Two files conflicted, both documentation. Your new GRE bullets went under Other Arista EOS Caveats where they belong, with the ArcOS section after them so the file stays alphabetical; the 26.08 release note took our entry into your skeleton.

Re-verified after the merge: 140 topologies, 93 transform, 47 refused as unsupported, zero hard failures. That includes the new ACL module — ArcOS refuses routing/30-acl-ipv4 and 31-acl-ipv6 cleanly with does not support 'acl' objects, and likewise the new routing/06-weight and 11-match-aspath, rather than rendering something wrong.

The 27 validation failures from the earlier breakdown are unchanged and still documented as caveats; none of them are config-phase failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs changes Use for PRs that need heavy changes before it's ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants